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1. Introduction 


* Image search engines (e.g. Google, Baidu) provide service of reverse image 
search, or search by image, to allow users to search for related images by 
uploading an image or image URL. 

* Reverse image search is a content-based image retrieval (CBIR) query 
technique that involves providing the CBIR system with a sample image 
that it will then base its search upon; in terms of information retrieval, the 
sample image is what formulates a search query.[1 | 
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End Of Monarch Butterfly Migration Could Be In Sight | Wisconsin 
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Monarch Butterfly Migration | Amusing Planet 
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Query image x EE ^ 
Search result 
Reverse image search may be used to 

* Locate the source of an image 

* Find higher resolution versions 

* Track down the content creator 

e Detect plagiarism [1]https://en.wikipedia.org/wiki/Reverse image search 
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Best guess for this image: petal 
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Climbing Hydrangea Problems | Garden Guides 
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500 x 334 - Sep 21, 2017 - The climbing hydrangea ("Hydrangea anomala petiolaris" has many 


similarities to the upright shrub nydrangeas. The main differences are leaf size and growth 
habit. The leaves of climbing hydrangea are round, shiny and about 3 inches wide. They will 
turn an attractive yellow in the fall. The white flowers ... 
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Plagiarism Detection 


The poster of TV play Fuyao released in June 2016, was accused of 
plagiarism. The original image is created in 2014 by LuHe, a painter. 
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No other sizes of this image found. 


Best guess for this image: illustration 
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Illustration Ltd, International illustrators and artists Agency, estbd 1929. Representing illustrators 
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Threat model 


* CBIR system can be a potential attack target. 


* Adversaries may evade the search engine by introducing 
perturbations to the query image. 


Climbing Hydrangea Problems | Garden Guides 
a agi /WWW. — com » Flowers v 
0 x 334 - Sep 21, 2017 - The climbing hydrangea ("Hydrangea anomala 
petiolaris '} has many similarities to the upright shrub nydrangeas. The main 
differences are leaf size and growth habit. The leaves of climbing hydrangea 
are round, shiny and about 3 inches wide. They will turn an attractive yellow 
in the fall. The white flowers 
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500 x 334 - Browse this user's interesting photos tagged BÆ, japan, 400d 
and more! Picssr is a new & refreshing way to browse Flickr photos 
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About 3,950,000 results (1.17 


Image size: 
1200 x 1600 


No other sizes of this image found. 


Best guess for this image: mid atlantic states 


Mid-Atlantic (United States) - Wikipedia 
https://en.wikipedia.org/wiki/Mid-Atlantic_(United_States) v 

The Mid-Atlantic, also called Middle Atlantic states or the Mid-Atlantic states, form a region of the United 
States generally located between New England and the South Atlantic States. Its exact definition differs 
upon source, but the region usually includes New York, New Jersey, Pennsylvania, Delaware, Maryland, 


Category:Mid-Atlantic states - Wikipedia 
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Category:Mid-Atlantic states. From Wikipedia, the free encyclopedia. Jump to: navigation, search. The 
main article for this category is Mid-Atlantic states. flag United States portal 
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BNY Mellon | The Investments Company for the World 
https://www.bnymellon.com/us/en/home.jsp v 

BNY Mellon shareholders elected Steven D. Black as a Director at the 2018 Annual Meeting of 
Stockholders held on Tuesday, April 10, 2018, effective immediately. With the addition of Black, BNY 
Mellon's Board will have 12 directors, 11 of whom are independent. READ MORE ABOUT OUR NEW BOARD 
MEMBER .. 


The Bank of New York Mellon - Wikipedia 
https://en.wikipedia.org/wiki/The Bank of New York Mellon v 

The Bank of New York Mellon Corporation, which does business as BNY Mellon, is an American worldwide 
banking and financial services holding company headquartered in New York City. It was formed on July 1, 
2007, as a result of the merger of The Bank of New York and Mellon Financial Corporation. It is the 

world's . 
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2. Technical Background 
CBIR framework 
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Local Features 


* Essentially, searching similar images in CBIR systems is a image 
descriptors matching process. SIFT (Scale Invariant Feature Transform ) [2] 
and SURF (Speeded Up Robust Features) [3] are widely used local 
featuring algorithms. 


SIFT descriptor matching 


e SIFT is more accurate while SURF has a higher speed. 


[1] Lowe, David G., and D. G. Lowe. "Distinctive Image Features from Scale-Invariant Keypoints." International Journal of Computer 
Vision 60.2(2004):91-110 

[2] Bay, Herbert, T. Tuytelaars, and L. V. Gool. "SURF: Speeded Up Robust Features." European Conference on Compute? Vision 
Springer-Verlag, 2006:404-417. 


SIFT [2] 


SIFT and SURF algorithms both build image pyramids, referred to 
as scale space, to find keypoints in different scale. They both apply a 
non-maximum suppression in a 3X3X3 neighborhood to find the 
potential keypoints. Point x is recorded as a keypoint only when 
abs(value(x))>T, while T refers to the threshold of the algorithm. 
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Scale 
(first 
octave) 


Difference of LT RT ST OR 
Gaussian Gaussian (DOG) 


Scale space 26 neighbors in 3x3x3 regions 


For each octave of scale space, the initial image is repeatedly 
convolved with Gaussians to produce the set of scale space. 
Adjacent Gaussian images are subtracted to produce the difference- 


of-Gaussian images. 
13 


SURE [3] 


Instead of using DOG-value, SURF uses the approximation 
of the determinant of Hessian matrix. Given a point x = 
(x, y) in an image I, the Hessian matrix H(x, o) in x at scale 
o is defined as follows 


Lxx (x, o) Lxy (x, ol 

Lyy(x%,o) Lyy(x,o) 

where Lxx(x,o)is the convolution of the Gaussian second 
order derivative with the image I in point x. 


Using box filter instead of Gaussian second order derivative 
to speed up, the approximation of L,,(x,a) is denoted by 
D. (x, o). 


Therefore det (Happrox) = DyxDyy — (0.9D yy) 


H(x,o) = 


Performance 


* CBIR systems perform well with rotated image, gray scale 
image, or even part of the image. 
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3. Methodology 


* Descriptor plays an important role in image 
matching. Therefore, the basic idea of bypassing an 
image retrieval system is to change the statistical 
histogram of keypoints descriptors. 


* Two methods 
(D Removing original keypoints from the image 
© Injecting external keypoints to the image 


* Note that utility-preserving is required. 


Utility-Preserving Requirement 


e An adversarial image should be utility-preserving, i.e., it 
must keep its original visual semantics for a human observer. 


+E 


Original image Adversarial image Adversarial image 
(acceptable) (unacceptable) 


3.1 SIFT Keypoints Removing Method 


Some SIFT keypoints removal algorithm have been proposed in resent 
years, We employ the fast and effective RMD [4] method to remove SIFT 
keypoints. 


RMD 


Original image Processed image 

187 SIFT keypoints 73 SIFT keypoints 
Removal with Minimum local Distortion attack (RMD) targets a limited 
number of keypoints to be erased. It introduce a value 6* > 0 that defines 
the subset £g« = {x:C < D(x) XC #6"). Erasing keypoint x in Sei 
means to decrease the absolute of DOG-value |D(x)| by an amount |6| such 
that its new value is below the threshold C ,C is the fixed contrast threshold. 


[4] Do, Thanh Toan, et al. "Deluding image recognition in sift- based cbir systems." ACM Workshop on Multimediais 
in Forensics, Security and Intelligence ACM, 2010:7-12. 


3.2 R-SURF:SURF Keypoints Removing Method 


As far as we know, there are no particular SURF keypoint removal algorithms 
in the early works. Therefore we propose a method called R-SURF to remove 
SURF keypoints, it is a optimization-based keypoints removing algorithm. 


For every target keypoint, support area S(x) is determined and three box filters 
Kyx, Kxy, Kyy are reconstructed based on the scale ø of the keypoint, and the 
following nonlinear optimization is constructed. 


min f = > leu, v)| 
(u,v)ES(x) 


(Ke BIO) x (Kyy@I'(x)) — 0.81 x (&., G9) < T 


uv) + elu, v) 20,(u,v) e S(x) 
I(u,v) + elu, v) € 255,(u,v) € S(x) 


Where l'(u,v) = I(u,v) + £(u,v) , the increment ¢(u,v) is added to each 
point I(u,v) in support area S(x). Try to find smallest distortion 3 £(u, v) when 
D '(x,o) < T holds. 


Original image Processed image 
186 SURF keypoints 122 SURF keypoints 


* This algorithm causes very little distortion to image. In general, we only 
remove some low-level, small-scale keypoints in the images to avoid 
obvious distortion. 
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3.3 Evaluation: Removal 


* Target system: 
The target system VisualIndex is a MATLAB and VLFeat based 
simple image indexing engine, created by Andrea Vedaldi. 


https://github.com/vedaldi/visualindex 
(codebook=10000, image database=flikr2500) 


Query in target system 


Visuallndex 


RMD with ten iterations 
Can bypass 


| The system can be bypassed with removal algorithm, when query 
| images contain small amount of keypoints. 


Evaluation: Removal 


Query in target system 


RMD with ten iterations 
Cannot bypass 


Query image Search result 


Query in target system 


RMD with sixty iterations 
Can bypass 
But cannot preserve utility Search result 


Query image 22 


3.4 Keypoints Injection Method 


In order to meet the utility-preserving requirement, keypoints 
injection methods are introduced as well. Two problems are 
needed to be addressed. 


* Where to inject SIFT keypoints? 
* Inject inside the original image directly 
* Inject to a frame surrounding the image 


* How to generate SIFT keypoints to be injected? 


* Generate SIFT keypoints with IMD algorithm (the inverse 
operation of RMD) 


* Construct a frame with malformed basic bricks 


IMD Algorithm 


IMD (Injection with Minimum local Distortion ) is the inverse operation of 
RMD algorithm. It targets a collection of potential keypoints, assigning 
location and scale information for each point randomly. 

For each potential keypoint, the absolute of DOG-value |D(x)| is calculated. 
If |D(x)| is smaller than the threshold C, a patch will be added upon the 
original image to meet the threshold. The size and value of the patch is 
calculated using scale information and Gaussian filter. 


Introduced keypoints 


Keypoints of original image — Keypoints of resultant image 


resultant image 
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Inject to a frame 


Original image 


IMD image 


Basic Bricks 


To minimize the size of frame, we construct the frame with basic bricks to 
generate keypoints as many as possible. 

In order to know the visual feature of SIFT keypoints, we extract SIFT keypoints 
as well as their surrounding regions from images, using K-means clustering 
algorithm to obtain multiple centroids, and finally build grains based on the 
result of clustering algorithm. We can see that a keypoint generates where there 
is an edge or corner. 
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grain 


Keypoints visualization 
A kind of grain is build. The contrast of grain should be changed to meet different SIFT 
threshold. Keypoints are generated as many as possible. 
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3.5 Evaluation: Injection 


Query in target system 


IMD inside image 
Can bypass 


Query image Search result 


The system can be bypassed with injection algorithm, when 
: query images contain small amount of keypoints. 


Evaluation: Injection 


Query in target system 


IMD inside image 
Cannot bypass 


Search result 


Query in target system 


Inject frame (IMD) 
Cannot bypass 


Query image Search result d 


3.6 Evaluation: Hybrid 


Query in target system 


Visuallndex 


RMD removal + IMD Injection 


Query in target system 


Visuallndex 


ad | = RMD removal + basic bricks Injection 


| This experiment shows that it is feasible to bypass 
| VisualIndex system with removal and injection algorithms. 29 
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4. Attacking Google Image Search Engine 


* Challenge 
* Unknown algorithm 
* 0/1 feedback 


* Our guess 


* Google image retrieval system uses local descriptors 
such as SIFT/SURF. 


* Bypassing strategy 
* Removal only 
* [njection only 
* Hybrid: Combination of the above two methods 


Removal Only 


G Google Search x e — x 
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About 2 results (0.74 seconds) 
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No other sizes of this image found. 


Best guess for this image: orange 


RMD with one iteration 
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Orange.co.uk has closed. But don't worry, get your Orange email, manage your account, and upgraW 
and more here. 


Orange : téléphones, forfaits, Internet, actualité, sport, video 
www.orange.fr/ v Translate this page 
Orange vous présente ses offres de téléphones portables, de forfaits mobiles et internet. Retrouvez aussi 
la messagerie, l'actualité, le sport, la video sur le portail Orange. 


Visually similar images 


3 mi ago 


Original image Adversarial image 


Report images 
e j Use precise location - Learn more 
Help Send feedback Privacy Terms 


For some images that contain very few keypoints, performing RMD : 
with one iteration can bypass google image search engine successfully. 41 


Removal Only 


Original image RMD with one iteration RMD with five iterations 
CANNOT bypass CAN bypass 


Keypoints removal algorithms may cannot remove big-scale keypoints 
without making unacceptable distortion. Experiment shows that if 
leaving the big-scale keypoints unremoved, 10% of keypoints can lead 
to correct searching results, due to the robustness of CBIR system. 
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Injection Only 


IMD on 15-pixel-wide frame IMD on 50-pixel-wide frame 
CAN bypass Google image search CANNOT bypass Google image search 


Injection only strategy performs well on some simple images, however, as 
the image on the right shows, sometimes ‘injection only’ strategy is useless 
and unacceptable. On account of utility preserving, we should make the 
adversarial image as natural as possible. Strategies ‘Removal only’ and 
‘Injection only’ may not meet this requirement sometimes. We should use a 
hybrid strategy for better performance. 
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Hybrid: Combination of Removal and Injection 


SIFT 
RMD with one iteration + IMD on 15-pixel-wide frame keypoints 


visualization 


Removed keypoints A 


New keypoints 
generated 


SIFT keypoints 
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Petal - Wikipedia 

https://en.wikipedia org/wiki/Petal ~ 

Petals are modified leaves that surround the reproductive parts of flowers. They are 
often brightly colored or unusually shaped to attract pollinators. Together, all of the 
petals of a flower are called a corolla. Petals are usually accompanied by another set 
of special leaves called sepals, that collectively form the calyx and lie 
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Petals are modified leaves that surround the reproductive 
parts of flowers. They are often brightly colored or 
unusually shaped to attract pollinators. Together, all of the 
petals of a flower are called a corolla. Wikipedia 
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Visually similar images 
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Successfully bypassing Google image search 
RMD with one iteration + IMD on 15-pixel-wide frame 
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No other sizes of this image found 


Best guess for this image: petal 


Petal - A Simple, No-Fee Credit Card 
https://www.petalcard.com/ v 
Build credit, track spending and manage money better, all with no credit score required 


Petal - Wikipedia 
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Visually similar images 


Successfully bypassing Google image search 
RMD with one iteration + basic brick frame 
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About 3,950,000 results (1.17 


Image size: 
1200 x 1600 


No other sizes of this image found. 


Best guess for this image: mid atlantic states 


Mid-Atlantic (United States) - Wikipedia 
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Category:Mid-Atlantic states. From Wikipedia, the free encyclopedia. Jump to: navigation, search. The 
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poster background. 0 0 1000*1000. Department of Paris art 


Koi | free png | images and psd downloads | pngtree 
, https://pngtree.com/so/Koi-0-1-0-1 v 
Tem 260 x 391 - PNG PSD. 2018 festive Chinese wind Festival theme Icon. 7 1 3000*3000. Beautiful . 
hand-painted color Koi, Hand, Beautiful, Four Koi PNG and PSD. PNG PSD. Beautiful hand- Í y 
painted color Koi. 0 0 600*600. Mid autumn illustration, Mid-autumn Festival, Illustration, Carp Que Image 
PNG and PSD. PNG PSD. Mid autumn illustration 


Search result: original image 38 


Evasion 


G Google Search x 


€ C | & 2 | https://www.google.com.hk/search?tbs-sbi:AMhZZiv7j : UxiEtAU.. 


57A9.jpg 


bank of new york mellon 


Google 2 


All Images 


About 2 results (0.71 seconds) 


Image size: 
1258 x 1658 


a m 
SL, 


Y 


No other sizes of this image found. 
DEFCON 


Best guess for this image: bank of new york mellon 


BNY Mellon | The Investments Company for the World 
https://www.bnymellon.com/us/en/home.jsp v 

BNY Mellon shareholders elected Steven D. Black as a Director at the 2018 Annual Meeting of 
Stockholders held on Tuesday, April 10, 2018, effective immediately. With the addition of Black, BNY 
Mellon's Board will have 12 directors, 11 of whom are independent. READ MORE ABOUT OUR NEW BOARD 
MEMBER 


The Bank of New York Mellon - Wikipedia 

https://en.wikipedia.org/wiki/The_Bank_of_New_York_Mellon v 
The Bank of New York Mellon Corporation, which does business as BI 
banking and financial services holding company headquartered in Ne! 
2007, as a result of the merger of The Bank of New York and Mellon Fil 
world's . 
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5. Discussion 


e Source/target attack 


* A source/target attack means to force the search result of an 
adversarial image to be a specific target image. 
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Pages that include matching images 
Climbing Hydrangea Problems | Garden Guides 
https: iei gardenguides com» Flowers v 
Lo B 300 x Sep 21, 2017 - The climbing hydrangea ("Hydr "ia a anon 
miteriles to the ide eee The main differences ar 


h abit The leaves of climbing hydrangea are round, shiny an d. abota 
turn an attractive yellow in the fall. The white flowers 


L_Dan's most interesting Flickr photos | Picssr 
FA Se benang dan/interesting?nsid-56758835 


334 - Browse this user's interesting photos tagged DE, japan, 
a new & refreshing w: vua se Flickr photos! 


Soen - HIM - RE | MRE Dees Imagict 
, mana com/ja/words/&£ v Translate this es, 
-i 00 x 334 n BEAS. /& X= murasaki/ (n) (1) 


color / violet / (2) type of soy sauce / (P) [&]purple nd - a pu 


Threat model 


specific target image y 


e Source/target attacks will support more sophisticated attacks. 
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Source/Target attacking demo 


Query in target system 


Injected with keypoints from 
target image 


Search result 


Query image 


This experiment shows it is possible to achieve source/target 
attack by injecting keypoints from target image. 

However, severe distortion is caused. 

We will seek for a better strategy in further study. 
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6. Conclusion 


* [n this work, we present 
* A threat model of evading CBIR systems 
e Several algorithms for removing/injecting keypoints 
* Three bypassing strategies: removal only, injection only and 
hybrid 
* Evading VisualIndex system 
* Evading Google Image Search Engine 
* To conclude, our work proves the existence of threats to 


CBIR systems and demonstrates that industrial-level image 
search engines, such as Google Image Search, are prone to 


be attacked with adversarial images. 


THANKS! 
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